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1. INTRODUCTION 

Website is the essential interaction tool between the organization and its users in today's digital era. 
Having a website allows organizations to attract users worldwide to access their information and gain a 
competitive edge over others [1]-[3]. Institutions that have a better level of website quality will pick up 
higher levels of client fulfillment leading to feasible competitive advantage [4]. Websites have become more 
pervasive now than in the past regardless of the sector (commerce, government administrations, educations, 
entertainment, health care, culture, industry, financial services, and many others) [5]-[7]. 

Universities put enormous effort into creating a valuable website to facilitate stakeholders to access 
information swiftly without coming to the institutions [8]-[13]. The quality of the university's website reflects 
the quality of universities activities [14]. Among the information available on universities’ websites include 
academic programs, facilities, infrastructure, activities, achievements, fields of experts, and so forth. Students 
have higher expectations with regards to website quality, starting from efficiency until the content [15]. 
Hence, quality websites can attract more users and increase their satisfaction. 

Generally, the term quality encompasses the capability of a service or product in terms of meeting 
the needs and expectations of the consumer [16]. Quality is described by International Organization for 
Standardization 9000 (ISO) as "the degree to which a set of inherent characteristics of an object fulfills 
requirements" (ISO, 2015). Quality also implies liberation from defects which means a release from errors that 
call for redoing the work related to field failures, customer claims, customer dissatisfaction and others [15], [17]. 
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Website quality is dependent on: to what extent it could meet the user's needs [18]. Additionally, it describes 
users assessment of website performing within delivering information based on its usage capability [19]. 

The term website quality differs from the traditional concept of quality because website quality 
should be perceived as a multidimensional factor that considers the users' and designers' aspects [20]. Despite 
experiencing a lot of weaknesses and problems in some universities websites, most universities are working 
hard to satisfy the international standard of website quality [21]. Evaluating the quality of a website is very 
important since the presence of university websites reflects the existence of a university [22]. So, it is 
imperative to study further on university website quality. Section 1 provides an introduction of the project. 
Section 2 outlines the research method. Section 3 discusses the results and finally, section 4 reports the 
conclusion and future works. 


2. RESEARCH METHOD 

Systematic literature reviews (SLR) is a common tool to synthesize the prevailing body of literature 
in a specific field [23], [24]. For this study, the SLR method was utilized to identify and synthesize related 
scholarly research papers [25]. SLR is defined as a method for identifying, evaluating, and synthesising the 
existing body of completed and recorded work created by researchers, scholars, and practitioners that is 
systematic, explicit, and reproducible. Contrary to the use of unstructured methods such as simple literature 
reviews that have a potential to be biased, SLR results are more reliable and tend to be unbiased [26], [27]. SLR 
comprises of three phases: review planning, review conduction, and result reporting [25] as shown in Figure 1. 
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Figure 1. The systematic literature review (SLR) method [15] 


2.1. Review planning 

The most important activities during the review planning phase are definition of the research 
questions for this SLR and creation of the review protocol. However, all other activities should not be 
neglected and should be taken seriously. The results of this phase should include a clearly defined review 
protocol containing the purpose and the procedures of the review. 


2.1.1. Research questions 

The first step in the SLR is to determine its focus by clearly frame the questions the review seeks to 
answer. The questions were made by following PICO table. PICO stands for population, intervention, 
comparison, and outcomes [17] The formula of each item in PICO are illustrated I n Table 1. 


Table 1. PICO summary 
PICO Formula 
Population University websites 
Intervention University website quality, evaluation factors, base model 
Comparison N/A 
Result The factors and indicators to measure the quality of university website, based model, evaluation problems 
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This SLR seeks to identify answers to the following research questions: 

RQ1: What are the problems for evaluating the quality of universities' websites? 

RQ2: What are the models used to evaluate the quality of universities' websites? 

RQ3: What are the factors and indicators being used to evaluate the quality of academic websites? 


2.1.2. Identifying the keyterms 

Based on the research questions mentioned above, the key phrase was "universities website quality 
evaluation." Then, the synonyms of the keyword ‘evaluation’ were identified which are ‘assessment’ and 
‘measurement’. The search string was formulated based on the main terms and their synonyms, and Boolean 
as shown: evaluation or assessment or measurement) and website and quality and (University or academic or 
education). The search keywords are used to find relevant studies in the paper’s title, keywords and abstract. 


2.1.3. Identifying the sources 

Five databases were selected for this SLR which include Institute of Electrical and Electronics 
Engineers (IEEE), Springer Link, Science Direct, Emerald, and Scopus. These databases were selected as 
they dispense the most important and the highest impact full-text journals and conference proceedings, 
related to university website quality. This SLR focuses on searching for scientific databases instead of 
specific books or technical reports, as it assumes that the major research results in books and reports are also 
usually described or referenced in scientific papers. 


2.1.4. Identifying the inclusion/ exclusion criteria 

To make certain that only relevant literature is accepted into the SLR, the inclusion and exclusion 
criteria is used. For the exclusion criteria: papers not in the English language; only the abstract but the full 
text is not available; does not have factors or characteristics. While for the inclusion criteria: full-text papers; 
review papers, and white papers published from 2010 to the year 2020 [28]. 


2.1.5. Identifying the data extraction strategy 

Data extraction is the process of capturing key characteristics of studies in structured and 
standardized form based on information in journal articles. The objective of this step is to design data 
extraction forms to accurately record the research’s information that are obtained from the selected papers. 
After implementing the inclusion/ exclusion criteria as illustrated in the previous step, information such as 
citation, the context of the study, base model, and the quality factors are extracted from the selected papers in 
order to answer the research questions. 


2.2. Review Conducting 
2.2.1. Identification of research 

The search process was separated into two stages which include primary and secondary search. In 
primary search, the papers were identified in the selected databases using the search string. In the secondary 
search, the selected primary studies' references were reviewed to determine any additional related studies. 
This process aims to ensure that the primary search phase has not missed any relevant literature. Table 2 
illustrates the results of the primary studies search. After passing the primary study search through five 
databases, 519 papers were selected. In the secondary stage, a backward search was conducted for the 
selected papers of the primary studies. 


Table 2. Primary search results 


Databases Number of papers Databases Number of papers 
Emerald 12 Springer 258 
IEEE 24 Scopus 151 
Science Direct 74 Total 519 


2.2.2. Selection of the studies 

This step narrows down the number of documents found in the previous searching phase. To begin 
with, eligibility criteria was applied to determine which of the studies identified in searches are pertinent 
based on the paper title while all irrelevant papers were discarded. The discardment was conducted by 
applying filters related to the university's website quality. Furthermore, the abstract of the filtered paper was 
assessed followed by the introduction and the conclusion/discussion of the filtered papers. This step consisted 
of a more thorough selection of the documents. 

After administering the first level of eligibility criteria, 13 primary studies associated with university 
website quality were retrieved. Using these 13 studies, a secondary search was carried out, which involved 
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reviewing the references in the selected primary studies to identify relevant supplemental studies. From this 
secondary search phase, 17 studies were collected from the references of primary search. The final number of 
selected studies was 30 studies. 30 studies went through the second level of inclusion/exclusion criteria 
where 28 studies were included and two studies were excluded due to duplication. 


2.2.3. Study quality assessment 

This step involves assessing the quality of the selected studies in order to support the extraction of 
the information for synthesis and result from analysis. A quality assessment checklist was adapted from [25]. 
Table 3 presents the checklist that was employed for assessing the paper quality. For the quality assessment, 
the following scale was used: Yes=1, Partially 0.5, and No=0 [29], it was concluded that the higher the study 
score, the greater the quality of the study which is concurrent with its ability to address the research question 
in a better way. 


Table 3. Study quality assessment checklist [13] 


No. Question Answer 
QAI! Are the aims clearly stated? Yes/No/Partially 
QA2 Are the methods used in each paper clearly described? Yes/No/Partially 
QA3 Do the objectives lead to conclusions? Yes/No/Partially 
QA4 Is the knowledge or understanding been extended by the research? Yes/No/Partially 
QAS Is the diversity of perspective and context been explored? Yes/No/Partially 
QA6 Are the links between data, interpretation, and conclusions are clear? Yes/No/Partially 
QA7 Does the detail/ depth/ complexity of the data is conveyed? Yes/No/Partially 


In the next step, the total scores for each article were calculated and the percentage was determined 
by dividing the total score by seven. A paper with the total score greater than five or 70% will be selected for 
the final study. From the previous 28 papers, four papers were rejected where three papers scored 4.0 or 57% 
and one scored 3.5 or 50%. So, there are 24 final studies selected from this SLR process. The result of quality 
assessment for the final selected studies is illustrated in Table 4. Figure 2 shows the overall process of this 
SLR process and the results for each steps. 


Table 4. Result of quality assessment for final studies 
Citations QAI QA2 QA3 QA4 Q5 QA6 QA7 Total % 
[8] 1 1 0.5 1 65 93 


= 


1 
[11] 1 1 1 1 1 1 1 7 100 
[30] 1 1 1 1 1 05 05 6 86 
[31] 1 1 1 1 1 1 1 7 100 
[32] 1 1 1 1 1 1 1 7 100 
[33] 1 1 1 1 1 1 0.5 65 93 
[34] 1 1 1 1 1 1 1 7 100 
[35] 0.5 1 0.5 1 1 05 05 5 7 
[36] 1 1 1 1 1 1 1 7 100 
[37] 1 1 1 1 0 05 05 5 7 
[38] 1 0.5 1 1 1 05 05 55 79 
[39] 1 0.5 1 05 1 05 1 5.5 79 
[40] 1 1 1 05 1 0.5 1 6 86 
[41] 1 1 1 1 1 1 1 7 100 
[42] 1 0.5 1 1 1 05 05 55 79 
[43] 1 1 1 1 1 1 1 7 100 
[44] 1 1 1 05 1 05 1 6 86 
[45] 1 1 1 05 1 05 1 6 86 
[46] 1 1 1 1 1 1 1 7 100 
[47] 1 1 1 1 1 1 1 7 100 
[48] 1 1 1 1 1 1 1 7 100 
[49] 1 0.5 1 1 1 05 05 55 79 
[50] 1 1 1 1 05 1 0.5 6 86 
[51] 1 1 1 1 1 1 0.5 65 93 


2.2.4. Data extraction and monitoring progress 

24 studies were selected and systematically reviewed. The extracted information such as citation, 
the context of the study, the base model, and the quality factors have been used in the final study and 
synthesized to answer the research questions. The main content from the extracted information represented in 
the Table 5. 
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Figure 2. Overall SLR process 


Table 5. Fields of studies and institutions for final papers 


ID Citation Field of Study Institutions 

R1 [8] University website Telkom University, Indonesia 

R2 [11] University website University of Kanjuruhan Malang, Indonesia 
R3 [30] Academic libraries Nigerian academic libraries 

R4 [31] University website Nigerian University 

R5 [32] University website Iranian state university 

R6 [33] Course website National Taiwan Ocean University 

R7 [34] University website Tanzanian Public University 

R8 [35] ollege library website Institute of Science and Technology, China 
R9 [36] University website Delft University of Technology, Netherlands 
R10 [37] University website Institut Teknologi Sepuluh Nopember Surabaya, Indonesia 
R11 [38] University website X University, USA 

R12 [39] University website University of Bari's, Italy 

R13 [40] University website 118 Portuga universities 

R14 [41] University website Malaysian Universiti 

R15 [42] University libraries Maharaja Sayajirao University 

R16 [43] University website Kenyan University 

R17 [44 Research institute X Agency 

R18 [45] University website Jember University, Indonesia 

R19 [46] Higher education services LL-Dikti IV 

R20 [47] University website Eastern Samar State University, Philippine 
R21 [48] University website Universities in Indonesia 

R22 [49] University website Universitas Sumatera Utara, Indonesia 

R23 [50] University website Three Jordanian universities 
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R24 [51] Language center websites Mulawarman University 


2.2.5. Data synthesis 

The quality factors were identified from the 24 relevant research based on user perspectives. Data 
coding technique was used to label and organize the factors into themes. From 24 studies, 79 quality factors 
were extracted. Based on the similarities and differences of data codes, a constant comparison was 
performed. To identify the similarities, filtration was done based on explicit and implicit duplication. Explicit 
duplication is the clear duplication using the same code. For example, code for ‘Usability’ that appeared in 
many papers will be included once. Implicit duplication is the difference code that has the same meaning. For 
example, ‘information quality’ and ‘content quality? have the same meaning. After removing duplications 
and applying memoing technique, six quality factors were generated as in Table 6. 
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Table 6. The list of universities websites quality factors 


No. Quality factors Study ID 
1 Information quality R10, R22, R8, R3, R9, R4, R5, R11, R13, R20, R1, R14, R19, R15, R16 
2 Specific content R10, R8, R9 
3 Usability R10, R22, R8, R3, R9, R4, R5, R11, R6, R12, R13, R1, R14, R19, R15, R7, R16 
4 Web appearance R10, R22, R3, R9, R20, R1, R14, R19 
5 Service interaction quality R3, R4, R5, R11, R6, R12, R13, R1, R15, R16 
6 Functionality R8, R4, R6, R12, R20, R1, R14, R7 


3. RESULTS AND DISCUSSION 
In this section, we talk about how this work can be helpful for research communities. This work also 
leads to some possible future work areas. This research was conducted to answer three main research questions: 


3.1. To investigate the issues of evaluating the quality of universities' websites 

Table 7 presents the issues of evaluating universities’ websites quality. The findings indicate that 
there is still a lack of study on university websites quality compared to business websites [33], [34]. From the 
developers aspects, some of the designers did not have the appropriate knowledge of usability engineering 
and user interface design which affect the quality of the websites. The most common issue is related to 
website evaluation aspects since website quality evaluation is complex and multidimensional and there is no 
specific evaluation model for university websites. The evaluation of universities' websites differs from other 
websites, in terms of structure and content. Besides, the factor of priorities and weights depend on the 
website type [11], [12], [52]-[54]. For example, user interface and atractiveness are essential factors to attract 
more consumers to an e-commerce website [55]. Conversely, information quality is more important for 
educational website [10], [56]. Moreover, there is often an overlapping among factors and sub-factors [57] 
which generally happens for the models inspired by software quality standards [5], [57]. For example, 
‘efficiency’ is the main factor in ISO 9126 model, but it is a subfactor in the WebQual 4.0 model. 
Additionally, several website quality evaluation models are not scalable in terms of indicators [5], [58]: The 
possibility to focus only on some quality factors or sub-factor while ignoring the others, is generally not kept 
into consideration [1], [59]. It is challenging to evaluate the quality of university website as various types of 
different indicators affect the website's design [60]. 


Table 7. The issues of evaluating universities websites quality 


Main Issues Issues Paper ID 
Researchers interest - Most of the research on website quality mainly focused on business websites R5, R6 
but lacks on university websites. 
Most of the research focused on the quality from developer and designer R1 
perspective with little work on user perspectives. 
Problems from Many university website designers did not have the appropriate knowledge of R2, R15 


website developers 
aspects 


Problems from 
website evaluation 
aspects 


usability engineering and user interface design. They caused unnecessary traffic 
on the internet and also inconsistent in terms of colors, structure, and fonts. 
Many university websites have problems in terms of accessibility, functionality, 
security, web appearance, and the quality of the content. 

Website quality evaluation is complex and multidimensional. 

There is no specific evaluation model for university websites that takes into 
account the needs of different user groups. 

The standard of evaluation website quality is still limited. 

General models have often been applied to assess the quality of academic 
websites, but they do not consider the requirements of specific stakeholders of 
the website. 

Limited academic research on the overall website quality and most of them 
concentrate on a specific topic on quality such as usability. 


R2, R3, R18, R21, 
R22 

R1, R2, R3, R4, R14 
R6 


R10, R15 
R6, R15 


R1, R3, R10, R15, 
R21 


3.2. To identify the base model or aspect for evaluating the quality of universities' websites 


Table 8 and Figure 3 illustrate the distribution of models or aspects of criteria used to evaluate the 
university website quality. There are 11 evaluation models were extracted from this SLR. The most frequent 
model used to measure the quality was WebQual 4.0 (46%), followed by ISO/IEC 9126 (13%). 


3.3. To investigate factors and indicators to evaluate the quality of universities' websites 

79 quality factors from 24 studies have been extracted in this SLR. After removing duplications and 
applying memoing technique, six quality factors were identified: Information quality, specific content, 
usability, web appearance, service interaction quality, and functionality. Besides the factors, this study also 
identified the indicators for each factors as listed in Table 9. 
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Table 8. Base model for website quality evaluation 


Base Model Study ID Base Model Study ID 
Aladwani's and Palvia's instrument R1, R5 QinUEM R14 
7Loci meta-model R15 Usability Heuristic guidelines R2 
CAWI questionnaire R13 WebQEM R9 
Functionalities features R3 Zhang Mei model R11 
ISO/IEC 9126 R6, R10, R17  WebQual 4.0 R4, R7, R8, R16, R18, R19, 
Performance criterias R12 R20, R21, R22, R23, R24 


Count of Citation 
Performance criterias 


4% 


Zhang Mei mode! 
4% 


WebQual 4.0 
46% 


WebQEM 
4% 


\ 


7Loci-meta-model 


4% Aladwani's and 


Palvia’s instrument 


9% 


4% 


4% 


13% 


4% 


CAWI questionnaire 


ISO/IEC 9126 


QinUEM 


Functionalities 
Features 


Usability Heuristic 
guidelines 
49 


Figure 3. Base model for website quality evaluation 


Table 9. List of factors and their indicators 


Factor Indicators Factor Indicators 
Information 1. Usefulness Web Appearance 1. Attractiveness 
Quality 2. Completeness 2. Changing look 
3. Clarity 3. Proper use of fonts 
4. Currency 4. Proper use of colors 
5. Information accuracy 5. Appropriate use of images 
6. Information believable 6. Proper use of multimedia 
7. Information relevant Te Style consistency 
8. Information detail 8. Proper choice of page length 
9. Information uniqueness 9. Good labeling 
10. Information broadness 10. Text-only option 
11. Information originality 11. Proper use of language 
12. Variety information 12. Good labeling 
Specific Content 1. Research interest & achievement 13. Text-only option 
2. Student portal 14. Enjoyment 
3. Course description 15. Noticeable logo 
4. Timetable 16. Logical structure 
5. Contact information Service Interaction 1. Good reputation 
6. General information Quality 2. Secure transaction 
7. Brochure 3. Secure information 
8. Admissions 4. Personalization 
9. Academic policies 5. Sense of community 
10. Research information 6. Responsiveness 
11. Frequently used forms 7. Trust 
Usability 1. Easy to learn to operate 8.  Recoverability 
2. Miscellaneous features 9. Availability 
3. Easy to navigate 10. Non-deficiency 
4. Easy to use 11. Additional services 
5. Competent Functionality 1. Searching and retrieving 
6. Creates a positive experience 2. Suitability 
7. Use of special plug-ins 3.  Student-oriented domain features 
8. Anonymity 4. Effective 
9. Browser sniffing 5. Accessibility 
10. Interactivity 6. Portable device compatibility 
11. Multi-language 7. Visibility features 
12. Page Speed 8. Bookmark facility 
13. Links visibility 9. Stability 
14. Alternative paths 10. Changeability 
15. User-Friendliness 11. Interoperability 
12. Flexibility 


Systematic literature review on university website quality (Ala’ Hasan Saleh) 


518 m) ISSN: 2502-4752 


4. CONCLUSION 

This paper employed SLR to search for papers related to university website quality by addressing 
the following research questions: RQ1: What are the problems in evaluating the quality of universities’ 
websites? RQ2: What is the base model used to evaluate the quality of universities' websites? RQ3: What are 
the factors and indicators being used to evaluate the quality of university websites? Regarding the issues for 
university website quality evaluation, a variety of evaluation criteria have been used. 24 final papers were 
considered eligible after the inclusion/exclusion process and quality assessment. Majority issues of university 
websites quality were related with website evaluation aspects since website quality evaluation is complex and 
multidimensional. Majority studies used the Webqual 4.0 as the base model to evaluate the universities 
website where information quality, usability, and service interaction quality are the factors for Webqual 4.0 
model. From the final articles, 79 factors were identified to evaluate the university website quality. After 
removing duplications and applying memoing technique, six quality factors were identified which are 
information quality, specific content, usability, web appearance, service interaction quality, and functionality. 
These identified website quality factors extend the factors used for Webqual 4.0 by adding specific content, 
web appearance, and functionality as the important factors to measure university website quality. With regard 
to future work, the researcher is interested to validate the factors and indicators identified from this SLR with 
experts in order to develop the university website quality model. 
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